method and system for indexing and searching contents of extensible markup language (xml) documents

ABSTRACT

A method and a computer system for indexing and searching the data content of nested field records, such as those in Extensible Markup Language (XML). The system includes an indexing and searching engine that constructs an improved full-text search index on the input XML data and then performs searches using the index. The system supports exact matches and partial matches using a wildcard character. The method transforms the input XML data into a form that encodes the data structural information by suffixing each word with its corresponding field qualifiers or an equivalent numerical pattern thereof. The resulting encoded words are then stored in a full-text index structure. Various types of full-index search may be performed. One alternative embodiment is to combine string matching and numeric or integer pattern matching to identify a particular word in a particular field. The portion of the word without field qualifiers is matched against the words in the index, and the pattern of numerals representing the word&#39;s field qualifiers is matched against the numeral patterns of the words in the index that correspond to their respective field qualifiers. Therefore, evaluation of complex field criteria is reduced to simpler and faster numeric matching.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Divisional of application Ser. No. 10/902,144,filed Jul. 30, 2004, which is a Divisional of application Ser. No.09/549,533, filed Apr. 14, 2000, now U.S. Pat. No. 6,782,380, issued onAug. 24, 2004, the entire contents of all of which are incorporated byreference.

FIELD OF THE INVENTION

The present invention relates to methods and systems to index and searchrecords stored in a language using nested fields, particularly thosestored in the Extensible Markup Language (XML). In particular, thepresent invention relates to an improved method and a computerizedsystem to index and search documents and data in languages such as XMLthat utilize nested fields.

BACKGROUND OF THE INVENTION

The Extensible Markup Language (XML) is a universally accepted formatfor representing structured data in textual form. It is widely adoptedin enterprise databases, and particularly in databases and applicationsconnected to the World Wide Web. The manipulation and exchange ofstructured data, e.g., spreadsheets, address books, financialtransactions, technical drawings, etc., is often challenging as the datais traditionally represented in platform or program dependent documentformats. XML provides a set of rules and guidelines for designing textformats for such data; these XML text formats are unambiguous,platform-independent, and extensible.

An example of a simple XML document is provided as follows: <record><name> <first_name> John </first_name> <last_name> Smith </last_name>123 Smith Drive </street> <city> New York </city> <state> New York</state> </address> </record>

Basic XML format includes tags with brackets, e.g., <city> begins afield and </city> ends a field. Thus, <city> New York</city> representsa field named “city” that contains the content “New York.” Fields can benested, e.g., “city” is an element in the field “address,” as shownabove. More complex syntax can be used for various types of data.

A key practical issue in realizing advantages afforded by XML is theneed for an efficient search method. Easy data manipulation and exchangerequires an effective method to handle computational intensive searchoperations for complex and concurrent queries, which are becoming commonplace in the use of networked enterprise databases and databasesconnected to the Internet.

Existing database management systems, such as relational database andobject-oriented database systems, are generally equipped with mechanismsor facilities for rapidly retrieving selected records based on keyfields in the database. Such facilities or mechanisms often depend uponthe data and the schema, and therefore are specific to each database. Avariety of complex data structures are implemented in databases tofacilitate fast retrieval of data based on key fields; examples includebinary trees, B-trees, and red-black trees. Additionally, various typesof indices are built for certain key words or fields that are frequentlyqueried in a database to enable fast searching on those words andfields.

Existing full-text indices allow rapid searches on any word in a body oftext. They are commonly used by Internet search engines such as Hotbotand Alta Vista to enable a user to quickly identify a particular Website. Although they vary considerably in their implementation, full-textindices essentially consist of a table of words in alphabetical order,with pointers or links to the corresponding locations of the words in adatabase or a file. Generally a full-text index also supports wildcard(represented by “*”) searches that locate words based on a partialmatch. For example, a search for “appl*” will find “apply,” “appliance,”etc.

Neither of these existing technologies provides an efficient way tosearch XML. Since XML represents structural data in a textual format, itlends itself only to a slow, sequential scan of the text in a search ofa particular record. Standard full-text indexing provides only anincomplete solution because the field context of each word is notpreserved. For example, a standard full-text index of the sample XMLdocument above supports a search for “Smith,” but not for “Smith” onlyin the “address” field. That is, one cannot locate an address with“Smith” in it using a full-index search; such a search will find allrecords in any field that has “Smith” in it. Some full-text indexingsystems have the ability to search for a word associated with aparticular property or field of a document (such as “Author is JohnSmith”), but this still does not provide a way to search based on thestructural context of a word in an XML file, which involves severalnested field qualifiers.

Therefore, much needed is an improved full text indexing mechanism forsearching XML data, which is capable of distinguishing between “Smith”in the last_name field and “Smith” in the street field, or between “NewYork” in the city field and “New York” in the state field. Such amechanism should also preserve information on nested fields, so that thestreet field is recognized as an element within the address field, andthe last_name field is recognized as an element of the name field. Thequeries such as “address contains New York” (search for any record thatcontains New York in the address field or any field under the addressfield) and “address/city contains New York” (search for any record thatcontains New York in the city field that is part of an address field)should rapidly retrieve the qualified records using such an improvedindexing and searching mechanism. To make fast and effective searchespossible, certain external data structures need to be constructed topreserve the inherent structure information in the XML data and toprovide a short cut to locate particular items.

However, the current state of the art only provides limited alternativesfor indexing and searching XML data. One approach is to create separateindices for each sub-fields, which preserves the structural informationof the data but drastically increases the overhead and therefore is notdesirable. Another approach is to use a directed graph to represent thenested fields. (Goldman R. et al., Lore: a database management systemfor XML, 2000) The search through a directed graph can be extremelycomputationally intensive and costly as the complexity of the data,hence complexity of the graph, grows. Both approaches result in an indexstructure whose complexity is comparable with that of the XML dataitself. A more efficient and cost-saving indexing and searching methodis desired.

SUMMARY OF THE INVENTION

To resolve the above problems, the present invention is directed to animproved method and a computer system for indexing and searching recordsin a language utilizing nested fields, such as XML. The presentinvention discloses an indexing and searching engine that constructs animproved full-text search index on the input XML data and then performssearches using the index. The indexing and searching engine according tothe preferred embodiment of this invention supports exact matches andpartial matches using a wildcard character.

In accordance with one aspect of the present invention, the methodtransforms the problem of indexing and searching nested field records,including XML data, into the problem of full-text indexing and searchingof plain text documents. The input XML data is changed into a form thatencodes the field structural information by suffixing each word with itscorresponding field qualifiers in their nested entirety, oralternatively, by suffixing each word with a numerical code pattern thatrepresents the word's corresponding field qualifiers in their nestedentirety. The resulting encoded words are then stored in a full-textindex structure.

In accordance with another aspect of the present invention, wildcardmatching may be used to perform searches with or without fieldqualifiers. To search using a wildcard without field qualifiers allowsidentifying a record including a particular word regardless the field ofthe record, whereas to search using a wildcard with field qualifiersallows identifying a record including a particular word in a designatedfield or fields that share certain level of similarly nested structure.

In accordance with yet another aspect of the present invention, acombination of string matching and integer pattern matching is used inthe search of a particular word. The portion of the word without fieldqualifiers is first matched against the words in the index, and then theword's field qualifiers are transformed into a pattern of numerals,e.g., integers, to be matched against the integer patterns of the wordsin the index that correspond to their respective field qualifiers.Therefore, evaluation of complex field criteria is reduced to simplerand faster numeric matching.

The present invention with all aspects of a method and computer systemfor indexing and searching nested field records, such as XML data anddocuments, significantly improves the effectiveness and speed of thesearch, and hence facilitates full realization of advantages of XML asan extensible, portable data exchange format.

Further features, objects, and advantages of the present invention areapparent in the examples and in the detailed description that follows.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Indexing

Though the present invention is described here with particular referenceto XML records, the present invention extends to any nested field recordwhere a series of fields and sub-fields are used to nest data from arecord.

According to a preferred embodiment of this invention, a nested fieldrecord, such as an XML document or any data stored in XML form, istransformed into a text of words that encode the field structure contextof each word in the XML data. The transformation is accomplished bygiving each word in the XML document a suffix that represents the fieldinformation. First, each field name is assigned a numerical code, suchas an integer. For example, the following encoding may be used in thesample XML document provided in the Background section: record = 1 name= 2 first_name = 3 last_name = 4 address = 5 street = 6 city = 7 state =8

Second, each word is assigned a suffix according to the numericalencoding of its field or nested fields. For example, in the sample XMLdocument, the first occurrence of “Smith” is found in therecord/name/last_name field, the numerical coding or pattern of thisnested field qualifiers is therefore “1/2/4,” and this record isrepresented as “Smith1/2/4.” Similarly, the second occurrence of “Smith”is found in the record/address/street field, the numerical coding orpattern is therefore “1/5/6,” and this record is represented as“Smith1/5/6.” Using the same encoding mechanism, the sample XML documentwould be transformed to the following text for indexing: John/1/2/3Smith/1/2/4 123/1/5/6 Smith/1/5/6 Drive/1/5/6 New/1/5/7 York/1/5/7New/1/5/8 York/1/5/8

And accordingly, a full-text index for this transformed data may bebuilt as follows: 123/1/5/6 Drive/1/5/6 John/1/2/3 Smith/1/2/4Smith/1/5/6 New/1/5/7 New/1/5/8 York/1/5/7 York/1/5/8

Depending upon the complexity of the XML data, deeply nested structuresmay be reduced to lists of words suffixed by longer numeral or integerstrings. There is no intrinsic limitation to this method; both thelength of the word lists and the length of the suffix string may grow.Therefore, the method of indexing according to the present invention ismuch more efficient, robust, and less computationally-intensive comparedto available methods such as building separate index for every field.

In an alternative embodiment of this invention, the field structurecontext of each word is encoded using strings of field names in thenested order. For example, as discussed above, the first occurrence of“Smith” is found in the record/name/last_name field, so this record maybe represented as “Smith/record/name/last_name.” Similarly, the secondoccurrence of “Smith” is found in the record/address/street field, andthis record may be represented as “Smith/record/address/street.” Usingthis encoding mechanism, the sample XML document would be transformed tothe following text for indexing: John/record/name/first_nameSmith/record/name/last_name 123/record/address/streetSmith/record/address/street Drive/record/address/streetNew/record/address/city York/record/address/cityNew/record/address/state York/record/address/state

And accordingly, a full-text index for this transformed data may bebuilt as follows: 123/record/address/street Drive/record/address/streetJohn/record/name/first_name New/record/address/cityNew/record/address/state Smith/record/address/streetSmith/record/name/last_name York/record/address/cityYork/record/address/state

The two alternative indexing methods according to the present inventionmay be used interchangeably for XML data of limited volume andcomplexity. However, when higher volumes of data with more complexnested field structures are involved, reducing the word suffixrepresentation to a numeral or integer string will both save disk andmemory space and decrease computational time for indexing and subsequentsearches. Searching

Once the encoded field qualifiers are stored in a full-text index alongwith each word, the content and the structure of the XML data arepreserved. Various full-text index searches may be performed to identifya particular word in a particular field or fields using the index.

According to one embodiment of the present invention, wildcard matchingmay be used to perform searches with or without field qualifiers. Tosearch for a particular word, e.g., “John”, without field qualifiers, awildcard character is added to the end of the word following thedelimiter “/”, e.g., “John/*”. This expression will match “John” in anyfield. To search for a particular word with field qualifiers, fieldqualifiers encoded in the indexing operation are used along withwildcard characters which represent unspecified fields. For example, inthe above sample XML document, two steps need to be completed to searchfor “New York” contained in the field “/record/address”. First,“/record/address” is transformed to the integer string “/1/5”, using thefield encodings established when the index was created. Because thesearch should also cover any fields that might be nested inside theaddress field, a wildcard character should be added at the end, e.g.,“1/5/*”. Second, the numeral pattern of the field qualifiers is appendedto each search term, e.g., “New/1/5” and “York/1/5”. This transformationconverts any field search into an equivalent plain text search.

By way of example, the following is a computer program segmentimplementing the method of encoding field qualifiers by a pattern or anarray of integers, according to one embodiment of the present invention.This function converts a field expression, such as “/record//name” intoa corresponding numeric array. A flag “fUseWildcards” is used to specifywhether a particular field expression in a search query may containwildcard characters. For example, the query “/record//name containsSmith” finds any field “name” within a field “record” that has “Smith”as the value of the “name” field. Therefore, a wildcard character isused between the field “record” and the field “name.” In other words,this expression should be able to also match “/record/patient/name”, and“/record/name”, etc. When a delimiter “/” is used at the beginning ofthe expression, such as “/record/name”, the “record” field is the toplevel field element. When there is no “/” at the beginning of theexpression, such as “record/name”, the “record” field can be nestedinside other fields. That is, “record/name” can match expressions suchas “/table/record/name” and “/customer/record/name.” Therefore, awildcard character should be used at the front of a field expressionwhen there is no delimiter “/”.

To perform the encoding, the input field expression is tokenized basedon the delimiter “/”. Each string token is then assigned to a numericvalue or identifier by calling the function “getFieldId.” If there is noidentifier returned, the token is inserted into the table of field namesso that a unique id can be created for the token by a separate function.void encodeFieldExpression(const char *expr, FieldIdList& fieldId, intfUseWildcards) { if (fUseWildcards) { if (*expr != ‘/’)fieldId.append(matchAny); else *expr ++; } DStringSet s;s.tokenize(expr, ‘/’, fUseWildcards); for (int i = 0; i < s.getCount( );++i) { const char *str = s.getString(i); if (strIsBlank (str))fieldId.append(matchAny); else { long id = getFieldId(str); if (id ==FAIL) { id = add(s.getString(i)); fieldId.append(id); } } }

By way of example, the following is a computer program segmentimplementing pattern-matching using numeral encoding of fieldqualifiers, according to one embodiment of the present invention.Suppose “a” is the numeric encoding of the field qualifiers of a word inthe index, and “b” is the numeric encoding of a field qualifier in asearch query. Each of the integers in the “a” and “b” arrays correspondsto a field name. The “b” array may contain wildcard characters so thatthe query will support words with similarly nested field structures. Forexample, as discussed above, the query “/record//name contains Smith”matches any field “name” within a field “record” that has “Smith” as thevalue of the “name” field. To match this expression, the “b” array wouldcontain: <record code>, <matchAny>, <name code>, where <record code> isthe integer corresponding to the “record” field, <matchAny> is awildcard character that matches any number of values, and <name code> isthe integer corresponding to the “name” field. static int lMatch(constlong *a, const long *b) { while (*a && *b) if ((*a = =*b) | | (*b = =matchOne)) { a++; b++; } else if (*b = = matchAny) { b++; if (!*b)return true; while (*a) { if (1Match(a, b)) return true; else a++; }return false; } else return false; if (*a) return false; if (*b) { if(*b != matchAny) return false; b++; if (*b) return false; } return true;}

A modified matching method is used in an alternative embodiment of thepresent invention. This method uses a combination of string matching andinteger pattern matching to identify a particular word in a particularfield or fields. First, the portion of the word absent the fieldidentifiers or their numeral encoding is matched against words in theindex, to identify the matched records. This is a typical word look-upused for a text search that is not limited by fields. Second, the fieldexpression of the search query is transformed into an numeral or integerpattern, which is then matched against the numerical encodingrepresenting field qualifiers of each word in the index. The resultingmatches are subsequently combined with the matches from the first step.For example, a search request “address/street contains Oak” may beconverted to the integer pattern (*,5,6,*) associated with the word“Oak”. The wildcard character at the beginning and the end of thepattern indicate that the address field may be inside another field andthat additional fields may be nested inside the street field. Therefore,evaluation of a complex field expression is reduced to a simple matchingof integer patterns. Replacement of string comparisons with numericalcomparisons accordingly improves the speed of the search.

Although alternative embodiments of the present invention have beendescribed in detail, it is to be understood that the same is by way ofillustration and example only, and is not to be taken by way oflimitation. Other modifications and variations that do not depart fromthe scope and spirit of the invention are understood to be a partthereof.

All references cited above are expressly incorporated herein to the sameextent as if each was individually incorporated by reference.

1. A method of creating a searchable index for a nested field recordcontaining data, comprising assigning at least one abbreviatedidentifier to at least one field of the record containing data, andcreating an index comprising a string that includes the abbreviatedidentifier matched with the data of said field.
 2. The method of claim1, wherein the nested field record is an XML record.
 3. The method ofclaim 2, wherein the abbreviated identifier is a numeral.
 4. The methodof claim 3, wherein the data of the XML record includes words.
 5. Themethod of claim 4, wherein the string of the index is organized asnumeral/word, wherein the numeral identifies the field and the word isthe data from the XML record.
 6. An index created by the method ofclaim
 1. 7. A method of searching for desired data contained in a nestedfield record, comprising searching the index of claim 5 for the desireddata and the matched abbreviated identifier.
 8. A computer readablemedium comprising instructions for carrying out the method of claim 7.